Add strong consistency check for data on disk #1604
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
vibhansa-msft
requested review from
souravgupta-msft,
ashruti-msft,
syeleti-msft and
jainakanksha-msft
as code owners
syeleti-msft
reviewed
Jan 9, 2025
|
@vibhansa-msft Please add the whole use case to validate the scenario also add the correct links in the PR. |
ashruti-msft
approved these changes
Jan 15, 2025
|
@vibhansa-msft , Do we need to update any public documentation for this new argument? |
jainakanksha-msft
approved these changes
Jan 15, 2025
vibhansa-msft
added a commit
that referenced
this pull request
Feb 3, 2025
* feat: support workload identity token (#1556) * feat: support workload identity token * Create block pool only once in child process (#1581) * create block pool in child only * Update golang.org/x/crypto to v0.31.0 (#1594) * Update golang.org/x/crypto to v0.31.0 * sync with main (#1603) * updated year in copyright message (#1601) * Use ListBlob for hns accounts (#1555) * Optimize HNS listing * Added statfs for block-cache (#1470) * Added statfs for block_cache * Add strong consistency check for data on disk (#1604) * Add strong consistency check for data on disk * bug in block cache open call (#1580) * current implementation of open file when opened in O_WRONLY will truncate the file to zero. This is incorrect behaviour. We don't see it in the normal scenario as write-back cache is on by default. Hence all the open calls with O_WRONLY will be redirected O_RDWR. To simulate this turn of the write-back cache and then open file in O_WRONLY. * Feature: Blob filter (#1595) * Integrating blob filter in azstorage * Serve getAttr call for destination file after the Copy finishes from the cache * Cleanup on start shall be set to cleanup temp cache (#1613) * Add Tests * Refactor the code and refresh the cache after copying the attributes * Automate blobfuse2 setup for new VM (#1575) added script for blobfuse setup and azsecpack setup in VM * * Update the Unit tests. * Refactor the Code * Update Changelog * do go fmt on src * Downgrade go version to 1.22.7 due to memory issues in 1.23 (#1619) * Enable ETAG based validation on every block download to provide higher consistency (#1608) * Make etag validation a defualt option * BUG#31069208: Fixed Prefix filtering from File Path (#1618) * Fixed the logic to filter out folder prefix from path * Added/Updated/Removed test case --------- Co-authored-by: weizhi <[email protected]> Co-authored-by: Sourav Gupta <[email protected]> Co-authored-by: Jan Jagusch <[email protected]> Co-authored-by: ashruti-msft <[email protected]> Co-authored-by: syeleti-msft <[email protected]> Co-authored-by: jainakanksha-msft <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
✅ What
Data being downloaded by block-cache is saved to local disk, if user choses to enable persistence. This feature is useful if application is reading the same files multiple times and we do not have enough in memory cache to hold all of them. When a block is downloaded it's saved on local disk. There are chances of accidental overwrite on the disk file which might corrupt the data. Later when same block is read by application Blobfuse will load the data from this disk file and end up serving corrupt data to user application.
Having a CRC check on the data when it was downloaded, protects us from those accidental corruptions.
🤔 Why
To provide strong data consistency checks and protect data persisted on local disk.
👩🔬 How to validate if applicable
If user choses to persist the downloaded content then this additional flag can provide and extra layer of security.
🔖 Related links
NA